A distributed data processing system typically allows for the sharing among locations or facilities of data processing functions by a collection of computers, and other devices linked together by a communications facility such as a network. Quite often, such distributed processing requires a highly structured environment, which allows hardware and software to communicate, share resources, and freely exchange information.
Developing program code for operating such distributed systems is difficult and time consuming. One of the most difficult tasks in the development of a distributed processing program is testing of the program. Many bugs in distributed processing programs either appear or disappear because of the relative timing of the events that occur at various nodes. With debugging programs, it is important to be able to reproduce erroneous behavior consistently. Distributed processing programs are inherently non-deterministic, and therefore make testing and debugging extremely difficult.
Typically, testing and debugging distributed processing programs involve running the several components of the program from controlled environments (e.g., debuggers) and causing the events in the system to occur in a controlled manner. Although this approach is effective, it is very time consming for regression testing. That is, making sure that the new "fixes" have not undone the "fixes" for the old bugs is very tedious with such an approach. Thus, such typical testing methods suffer from the lack of a deterministic timing behavior and an automatic means for regression testing.
Therefore, there is a need in the art for a testing technique for distributed processing programs that is automatic, deterministic, and allows for regression testing.